System and method for piece picking or put-away with a mobile manipulation robot

ABSTRACT

A method and system for piece-picking or piece put-away within a logistics facility. The system includes a central server and at least one mobile manipulation robot. The central server is configured to communicate with the robots to send and receive piece-picking data which includes a unique identification for each piece to be picked, a location within the logistics facility of the pieces to be picked, and a route for the robot to take within the logistics facility. The robots can then autonomously navigate and position themselves within the logistics facility by recognition of landmarks by at least one of a plurality of sensors. The sensors also provide signals related to detection, identification, and location of a piece to be picked or put-away, and processors on the robots analyze the sensor information to generate movements of a unique articulated arm and end effector on the robot to pick or put-away the piece.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/728,080, filed on Oct. 9, 2017, now U.S. Pat. No. 9,904,604, which isa continuation of U.S. patent application Ser. No. 14/340,896, filed onJul. 25, 2014, now U.S. Pat. No. 9,785,911, which claims the benefit ofprior U.S. Provisional Patent Application Ser. No. 61/858,590, filed onJul. 25, 2013, the entire contents of which are hereby incorporated byreference.

TECHNICAL FIELD

This invention relates generally to supply chain, manufacturing andlogistics automation equipment systems. More specifically, the presentinvention is directed to systems, devices and methods useful for thepurpose of automatically picking items from, and replacing items to, astorage location that uses common infrastructure such as racks orshelves.

BACKGROUND

In logistics facilities, such as distribution centers or retail stores,goods are stored for retrieval by a pick worker or customer. Each typeof item is known as a Stock Keeping Unit (SKU), and each SKU has aspecific location in which it is kept. These items can be stored openlyon shelving racks, or in compartmentalized containers, such as boxes orbins.

In a wholesale center, items are often stored in sealed cases, whereindividual units are packed together in a shipping case, as when theyare received from a manufacturer. Cases may further be grouped togetherand stored on pallets, which is common for freight shipments of goods.

When goods need to be retrieved individually for order fulfillment orselection by a customer, they are typically stored individually and arenot grouped into cases or pallets. The process of breaking the cases orpallets for individual product picking, that is, taking the individualpieces from the case or pallet and placing them in a specific storagelocation in a facility, is called put-away. The process of picking orselecting individual items from a specific storage location in afacility is known as piece picking or each-picking. Put-away and piecepicking happens in both distribution warehouses and retail centers,whereas case-picking or pallet-picking typically only happens at awholesale distribution center.

A fundamental problem with piece picking, and to a lesser extentput-away, is that it is inherently time consuming; it requires asignificant portion of time to be spent traveling from one item storagelocation to another. For put-away a person manually brings product casesto the pick locations and breaks them open to facilitate piece picking.For piece picking of a product there is the added time it takes to findand identify the specific item of interest in its unique storagelocation. This is often accomplished by specific SKU numbers thatpositively identify the item to be picked. While different SKUs mayappear to be the same, there may have some internal variations, such asweight, which cannot be identified outwardly. Finally, a person mustmanually pick or grasp the item and transfer it into a transportcontainer, such as a cardboard box or plastic tote for shipping.

Due to the time consuming nature of piece picking it is a very costlymanual process, and therefore has received much attention byorganizations looking to save time and money. There are many solutionsfor both optimizing and automating various aspects of piece picking.Some techniques look to minimize the amount of travel time required tomove from one point to another by reorganizing the SKU locations suchthat the most frequently accessed items are grouped together or requirea minimum amount of reach by a worker grasping the item.

Automation solutions range from augmenting manual labor with varioustechnologies to completely replacing labor with customized pickingequipment and infrastructure. For example, some automation systemssupport manual workers with barcode or radio frequency identification(RFID) scanners that enable them to more rapidly locate and identify aproduct. Others, such as voice picking technology, provide the piecepicker with an audio and speech interface headset that communicates tothe worker which items to pick and their location thereby enabling ahands-free process that improves speed and productivity.

There are also many types of automated machines that enable moreefficient picking operations. For example, large scale goods-to-personAutomated Storage and Retrieval Systems (AS/RS) allow a pick worker toremain in a fixed location. These systems have movable SKU storage binsthat can be carried by a machine to and from a fixed storage locationand delivered to a worker for picking individual pieces out of the bins.There are also Automated Guided Vehicle (AGV) systems that can transferstorage racks to and from a pick area where a worker can locate and grabthe requested item.

The automation equipment technologies presently available for pickingoperations require a substantial modification of infrastructure for thelogistics center in which they are used. This requires a significantup-front investment from the facility, which is difficult to afford andis the main reason such solutions have not been widely adopted. As such,many distribution facilities still rely on manual labor to accomplishpiece picking. Further, current automation systems are not viable forretail centers because the infrastructure must also be accessible to thecustomer. That is, current automation equipment cannot be used within aretail facility which relies on simple static shelving for productstorage and display.

Currently, logistics facilities follow a standard process for put-awayand picking of goods. Items arrive into the facility at a receivingarea, typically in cases or pallets, and are commonly registered into anInventory Management System (IMS) or Warehouse Management System (WMS).A WMS is a software database which stores information about SKUs whichmay include the product size, weight, inventory count, storage location,etc. After the items are received, they are put-away into their storagelocations, typically open shelving or racks. This is usually a manualprocess which involves a stock worker physically moving the items to alocation and transferring the items onto the shelf or rack.

Picking is done by a manual pick worker, also called selector or picker,in a warehouse, or by a customer in a retail facility. In a warehouse,picking happens after an order is received from an external customer.The orders are typically registered with the WMS, which then creates awork order, commonly known as a pick list, which instructs the pickerwhich items must be retrieved, their quantities, and location within thefacility. The picker then must find the items and physically transferthem to a shipping container that is associated with the order.

The two primary objections to automation for picking using currentlyknown systems are: first, that the perceived upfront cost is too high,and second, that automation equipment is not flexible enough toaccommodate changes to inventory or the operation process. As such, themajority of businesses have continued to rely on manual picking labor.The high cost and inflexibility of current automation is largely due tothe infrastructure changes required for such solutions. Therefore, asolution that does not require changing significant infrastructure in afacility, such as using existing shelving and racks, and worksside-by-side with manual labor is desired. Such a solution would reduceupfront cost and keep the flexibility of human workers available.

SUMMARY

The presently disclosed invention overcomes many of the shortcomings ofthe prior art by providing systems, devices and methods for roboticpiece picking or put-away directly from existing stock item locations ina logistics facility. The presently disclosed invention provides amobile robotic system that includes sensors and manipulator arm(s) toperceive, localize, reach, grasp and transfer SKUs from a storage rackto a transport container for piece picking, or conversely, from atransport container to a storage rack for put-away. This system andmethod allows existing facility infrastructure to remain intact andfurther allows the facility to use both manual picking and roboticpicking interchangeably.

The presently disclosed robotic system and method solves several aspectsof robotic piece picking or put-away which are challenging and remainunsolved in the prior art. Specifically, the system enables roboticpicking to be done rapidly using a high degree-of-freedom manipulatorarm on a mobile base that can autonomously navigate and position itselfwithin an existing facility. The method and unique system design enableperception, localization and grasping of SKUs in a sufficiently fastmanner that is essential for operational viability and economy. It alsoreduces the complexity and cost required for autonomous navigation ofthe mobile base.

According to its major aspects, and briefly stated, the presentlydisclosed invention includes a system for piece picking or put-awaywithin a logistics facility comprising a central server and at least onemobile manipulation robot. The logistics facility may be a warehouse,distribution center, manufacturing facility, or retail facility. Thecentral server comprises a server communication interface, one or moreserver processors, and a server memory. Each of the mobile manipulationrobots comprise a mobile base, at least one articulated manipulator armhaving an end effector, at least one piece containment area, a pluralityof sensors, a remote communication interface, a robot memory configuredto store robot specific information, and one or more robot processorscoupled to the sensors, the robot memory, the mobile base, and the atleast one articulated manipulator arm.

The robot specific information may include at least calibration data forthe plurality of sensors. The robot memory may comprise computer programinstructions executable by the one or more robot processors to receivedata from and send data to the central server, process data receivedfrom each of the sensors, and output control signals to the mobile baseand the at least one articulated manipulator arm.

Further, the plurality of sensors provide signals related to detection,identification, and location of the piece to be picked, and the one ormore robot processors analyze the sensor information to generatearticulated arm control signals to guide the end effector of the atleast one articulated manipulator arm to pick the piece. The sensors mayalso provide signals related to a unique identification for the piece tobe picked, an obstacle detected in the path of the at least one mobilemanipulation robot, and a current location within the logistics facilityof the at least one mobile manipulation robot.

In certain embodiments of the system, the server memory may comprisecomputer program instructions executable by the one or more serverprocessors to receive data from a warehouse management system anddispatch the at least one mobile manipulation robot. The servercommunication interface may connect with the remote communicationinterface to send and receive piece picking data which may include aunique identification for each piece to be picked, a location within thelogistics facility of the pieces to be picked, and a route for the atleast one mobile manipulation robot to take within the logisticsfacility. The unique identification for the piece to be picked maycomprise a shape of the piece, a size of the piece, a weight of thepiece, a color of the piece, a property of the construction material ofthe piece, such as roughness, porosity, and deformability, a visualmarking on the piece, a barcode on the piece, or any combinationthereof. Further, the connection between the server communicationinterface and the robot communication interface may be via one or morewired or wireless networks, or a combination thereof.

In embodiments of the system, the at least one mobile manipulation robotmay be able to autonomously navigate and position itself within thelogistics facility by recognition of at least one landmark by at leastone of the plurality of sensors. The landmark may be a verticallymounted marker placed at a specific location within the logisticsfacility, or may be other identifiable visual or audible landmarkswithin the logistics facility. The sensors may be any 3D device capableof sensing the local environment such as, for example, 3D depth cameras,color cameras, grey scale cameras, laser ranging devices, sonar devices,radar devices, or combinations thereof.

In embodiments of the system, the at least one piece containment areamay be configured to sense a weight for a piece placed therein. The atleast one piece containment area may be at least one of a platform, apick-to-kit holder, a container holder, or any combination thereof.Further, more than one piece may be placed on the at least one piececontainment area by the articulated manipulator arm.

Certain embodiments of the system may further comprise a conveyancedevice configured to accept pieces from the at least one mobilemanipulation robot. The conveyance device may be a conveyor belt whichtransfers the accepted pieces from a transfer area to a receiving area,wherein the receiving area is a packing area, a shipping area, a holdingarea, or any combination thereof.

In certain embodiments of the system, the at least one mobilemanipulation robot may further comprise a user interface having agraphical display monitor and an input device. The input device may be atouch screen, voice command interface, facial tracking interface, smallliquid crystal display (LCD) interface, track ball, or keyboard.Further, the user interface may display user information such asdirection indicators showing the intended direction of movement of therobot, and error information. In embodiments, the mobile manipulationrobot may also include at least one safety light, an alarm buzzer, andat least one emergency stop button reachable by nearby workers.

In certain embodiments of the system, the at least one articulatedmanipulator arm may have a first end portion pivotally carried by thewheeled mobile base of the robot and a second end portion comprising theend effector. Further, the first end portion of the at least onearticulated manipulator arm may be mounted on a vertical actuator stageconfigured to raise or lower the at least one articulated manipulatorarm. The end effector may be a gripper, a suction cup, anelectroadhesion end effector, a magnetic end effector, or combinationsthereof, and the robots may comprise an end effector swap mechanismconfigured to permit a change of the end effector. When the end effectoris a suction cup, such may be connected to a vacuum pump through avalve, wherein actuation of the valve may be controlled by the one ormore robot processors.

In certain embodiments of the system, the at least one articulatedmanipulator arm may further comprise an extension tool positioned at ornear the second end portion. The extension tool may be sized so that itcan fit into a shelf without the risk of obstructing the view of thesensors and/or interfering with non-picked items on the shelf. Incertain embodiments, the extension tool may be long enough to reach intothe back of a shelf to allow the end effector to pick an item placedtherein, and may have a diameter that is smaller than the diameter ofthe end effector.

In certain other embodiments of the system, at least one sensor may bepositioned at a central point on the at least one articulatedmanipulator arm such that rotation of the at least one articulatedmanipulator arm directs the at least one sensor to view the at least onepiece containment area. The at least one piece containment area maycomprise a calibration target which allows calibration of the at leastone sensor located at the central point on the at least one articulatedmanipulator arm.

In certain embodiments of the system, the at least one mobilemanipulation robot may further comprise batteries and/or a charging portfor connection to a charging station. Such charging may be accomplishedmanually by a wired connection to warehouse power, or automatically viaa charging port or station.

The presently disclosed invention also includes a method of piecepicking or put-away within a logistics facility. The method may use asystem for piece picking or put-away which includes a central server andat least one mobile manipulation robot as defined in any of theembodiments described above, and may include the steps of:

receiving at a central memory via a server communication interface atleast one piece picking order including at least one item to be picked;

generating at the one or more server processors a piece pickingitinerary based on the at least one piece picking order, wherein theitinerary includes a unique identification for each item to be picked, alocation within a logistics facility of the items to be picked, and aroute for the at least one mobile manipulation robot to take within thelogistics facility;

receiving at a robot memory the piece picking itinerary;

moving the at least one mobile manipulation robot along the route to thelocation within the logistics facility of the items to be picked;

picking the at least one item to be picked from the location using anend effector of at least one manipulator arm; and

placing the at least one item to be picked in the at least one piececontainment area.

In certain embodiments of the method, the plurality of sensors mayprovide signals related to detection, identification, and location ofthe at least one item to be picked, and one or more robot processors mayanalyze the sensor information to generate articulated arm controlsignals which guide the end effector to pick the item. Furthermore,these sensors may allow the at least one mobile manipulation robot tomove through a logistics facility autonomously by recognition of atleast one landmark.

In embodiments of the method, the at least one piece picking orderreceived at the central memory may be generated by a warehousemanagement system. In other embodiments, a piece picking order may begenerated by a human user at a user interface which is attached to therobot and which communicates with the one or more robot processors and arobot memory. In such an embodiment, the piece picking itinerary may begenerated by the one or more robot processors, or may be generated bythe one or more server processors after the piece picking order has beensent to the central memory via communication between a robotcommunication interface and the server communication interface.

In certain other embodiments of the method, at least one sensor may bepositioned at a central point on the at least one articulatedmanipulator arm such that rotation of the at least one articulatedmanipulator arm directs the at least one sensor to view the at least onepiece containment area. The at least one piece containment area maycomprise a calibration target which allows calibration of this at leastone centrally located sensor.

Furthermore, the method may improve pick accuracy over the prior art bysensing a weight for the at least one item placed in the at least onepiece containment area.

BRIEF DESCRIPTION OF DRAWINGS

Aspects, features, benefits and advantages of the embodiments hereinwill be apparent with regard to the following description, appendedclaims, and accompanying drawings. In the following figures, likenumerals represent like features in the various views. It is to be notedthat features and components in these drawings, illustrating the viewsof embodiments of the present invention, unless stated to be otherwise,are not necessarily drawn to scale. The illustrative embodiments in thefollowing drawings are not meant to be limiting; other embodiments maybe utilized and other changes may be made without departing from thespirit or scope of the subject matter presented herein.

FIG. 1A and FIG. 1B are the front and side views, respectively, of anexemplary embodiment representing a mobile manipulation robot inaccordance with certain aspects of the presently disclosed invention.

FIG. 2 is a block diagram of an exemplary embodiment representing therobotic system with the hardware and software modules of a centralserver and a plurality of mobile manipulation robots in accordance withcertain aspects of the presently disclosed invention.

FIG. 3 is a simplified overhead floor plan diagram of a representativelogistics facility.

FIG. 4 is a diagram depicting an exemplary embodiment of mobile robotnavigation using visual landmark location markers in accordance withcertain aspects of the presently disclosed invention.

FIG. 5A and FIG. 5B are the top and front views of an exemplary picklocation with example pieces, grasp positions and vectors in accordancewith certain aspects of the presently disclosed invention.

DETAILED DESCRIPTION

In the following description, the present invention is set forth in thecontext of various alternative embodiments and implementations involvinga system and method for automated robotic piece picking or put-awaywithin a logistics facility, where the logistics facility may be, but isnot limited to: a warehouse, a distribution center, a manufacturingfacility or a retail facility. The presently disclosed inventionutilizes both robotics hardware and software technologies that aredetailed in the following description.

The above summary and drawings are not intended to describe or show eachillustrated embodiment or every possible implementation of the presentlydisclosed invention. Furthermore, various aspects of the system andmethod for piece picking or put-away with a mobile manipulation robotmay be illustrated by describing components that are coupled, attached,and/or joined together. As used herein, the terms “coupled”, “attached”,and/or “joined” are interchangeably used to indicate either a directconnection between two components or, where appropriate, an indirectconnection to one another through intervening or intermediatecomponents. In contrast, when a component is referred to as being“directly coupled”, “directly attached”, and/or “directly joined” toanother component, there are no intervening elements shown in saidexamples.

Relative terms such as “lower” or “bottom” and “upper” or “top” may beused herein to describe one element's relationship to another elementillustrated in the drawings. It will be understood that relative termsare intended to encompass different orientations of aspects of thesystem in addition to the orientation depicted in the drawings. By wayof example, if aspects of the mobile manipulation robot shown in thedrawings are turned over, elements described as being on the “bottom”side of the other elements would then be oriented on the “top” side ofthe other elements as shown in the relevant drawing. The term “bottom”can therefore encompass both an orientation of “bottom” and “top”depending on the particular orientation of the drawing.

As defined herein a Stock Keeping Unit (SKU) refers to a distinct item,and embodies attributes associated with the item that may distinguish itfrom another item. For a product, these attributes may include, but arenot limited to, the product manufacturer, product description, material,size, shape, color, weight, and packaging. Further, an individual SKUmay also have a code imprinted thereon which may indicate some of thesame above attributes. Examples of such codes include at least barcodessuch as a Universal Product Code (UPC), International Article Number(EAN), and Global Trade Item Number (GTIN).

Referring now to the drawings, embodiments of the system and method forpiece picking or put-away with a mobile manipulation robot are shown inFIGS. 1-5. FIGS. 1A and 1B are front and side views of an exemplaryembodiment of one of a plurality of mobile manipulation robots 100 thatcan be used within the system. Internal details of components andsoftware relevant to the system are shown in FIG. 2, which provides ablock diagram of an exemplary embodiment of the system comprising acentral server 200 and a plurality of mobile manipulation robots 100.The server may have an electronic communications interface (servercommunication interface 240) that connects with an electronicscommunication interface on the manipulation robot(s) (remotecommunication interface 210). This connection may be established througha wireless network via a wireless access point in a preferredembodiment. Other embodiments may include or instead use a differenttype of communication, such as a tethered wire connection or otherpoint-to-point wireless data exchange.

As shown in FIGS. 1A, 1B and 2, in certain embodiments the individualmanipulation robots 100 may have a wheeled mobile base 160, internalbatteries 190, and an onboard computer processor 218 with memory storage216. The robots may also have at least one temporary storage bed 140 forpicked items and at least one robotic manipulator arm 120. The onboardcomputer processor 218 may be configured to run a set of programs withalgorithms capable of performing navigation and picking. Further, theonboard computer processor 218 utilizes data from sensors (150, 110), tooutput control signals to the mobile base 160 and manipulator arm 120for navigation and picking, respectively.

As mentioned above, the onboard computer processor 218 may also havelocal persistent memory storage 216 which stores specific informationrelevant to the configuration of each manipulation robot 100. Suchinformation may include sensor calibration data, actuator tuningparameters, and other platform specific data. The onboard computerprocessor 218 may also communicate with the central server 200 toreceive pick order information and respond back with confirmation datato inform the central server 200 of successful picks or any errors thatmight occur.

Each manipulation robot 100 may also have a user interface 130, whichincludes a graphical display monitor and an input device, where theinput device may be a touch screen 130, a track ball, voice command, akeyboard, input buttons or any combination of these devices and possiblyothers. The user interface 130 allows a user to command and control eachmanipulation robot 100 to perform localized tasks and to enter productpicking dispatch information manually, thus sending the robot on itsmission. In addition, in one embodiment, each manipulation robot 100 maycontain an external swappable memory port on a side, where necessaryinformation may be uploaded to the robot directly when the operatorinserts a data storage device, thus by-passing the wirelesscommunication to the server. The data storage device may be a disk, USBflash device, or other forms of external memory storage devices. Inother embodiments, the data is transferred through proximitycommunication technologies, such as near field communication (NFC),Bluetooth, or short-range radio-frequency identification (RFID)standards.

Each manipulation robot 100 may also be equipped with safety featureswhich may include: one or more safety lights or strobes 155, an audiblewarning annunciator or horn, one or more emergency stop buttons 157, theability to display fault, error and/or intended action (such asnavigation turn signal) information on the user interface 130 or at someother point on the manipulation robot 100, or any combination thereof.

Furthermore, each manipulation robot 100 may be configured to receivesignals from the central server 200, or directly from the WMS 201, whichmay indicate an emergency and may direct the robot 100 to stop and/ormay further activate the one or more safety lights or strobes 155 and/oraudible warning annunciator or horn. In the event that an unstableand/or unsafe diagnostic state for the manipulation robot 100 isdetected by the one or more robot processors 218, the robot 100 may bestopped. The manipulation robot 100 may also be stopped if the sensors(150, 110) detect a human or obstacle in close proximity, or detectunsafe operation of the robot 100. Such signals may be processes at thecentral server 200 which may then control the robot speed and ordirection of operation.

An articulated robot manipulator arm 120 is used in the presentlydisclosed system to pick pieces with the common variability found initem size, shape, weight and placement within a logistics facility. Anexemplary representative drawing of such variability is shown in FIGS.5A and 5B, which are the top and front views, respectively, of a stockedshelf 500. Common logistics storage infrastructure, such as an ordinaryshelf or rack 500, does not constrain any item location and orientationfor the purposes of any deliberate accuracy, therefore, in order for arobot to do a pick at random it must have sufficient freedom to grasp anitem in various configurations. As such, a high degree-of-freedom robotmanipulator arm 120 provides the manipulability necessary to pick anitem in any configuration in which it is found.

The manipulation robot 100 may physically adjust for variations in itemlocation and orientation which may be determined from sensor 110information prior to the pick. In a preferred embodiment, themanipulator arm 120 may be mounted to the robot frame 125 at a positionon top of the mobile base 160 of the manipulation robot 100. Themanipulator arm 120 enables the robot to reach multiple shelf levelheights. Some embodiments may extend the vertical reach of a robot bymounting the manipulator arm 120 on a vertical actuator stage. Thevertical actuator stage would be able to raise and lower the manipulatorarm 120 so an end effector 175 can reach both higher and lower picklocations. In different embodiments of the system, additional highdegree-of-freedom robot manipulator arms 120 may be included which mayprovide additional lift capability to pick objects of various shapes andsizes when the arms work cooperatively, or to pick more than one objectat a given pick location using arms working in parallel butindependently. For multi-arm embodiments, the arms may be the same orhave different kinematic configurations, and may have the same ordifferent end effectors.

The present robot system uses a grasping end effector 175 on themanipulator arm 120 to pick items from their stored location andtransfer them to a temporary location, or vice-versa. In a preferredembodiment, the grasping end effector may be a suction cup 175, whichmay be connected to a vacuum pump through an onboard computer processor218 controlled valve. The vacuum suction at the suction cup 175 may beengaged and disengaged by actuating the valve, thereby allowing themanipulation robot 100 to grasp the desired pick item on contact andthen release it when necessary. The use of a suction cup 175 also allowsthe robot to grasp a target piece at a single point and orientation,which reduces the computation time required for the system to determinehow to grasp the pick item.

In other embodiments, the end effector may be a mechanically actuatedgripper such as, for example, a robotic hand having articulated digits.In yet other embodiments, the end effector may be a simple gripper, anelectroadhesion end effector, a magnetic end effector, or combinationsthereof, and the robots may comprise an end effector swap mechanismconfigured to permit a change of the end effector. Exemplary magneticend effectors may utilize electromagnets, permanent magnets, or magnetarrays which provide opposing magnetic fields. An electroadhesive endeffector may use reversible electrostatic adhesion to grip an item whileit is picked and put. In embodiments which use an electroadhesive ormagnetic end effector, such end effectors may be powered by anelectrical power supply configured to generate an electrostatic ormagnetic adhesive force that is used to releasably adhere the item tothe end effector. The onboard computer processor 218 may control thepower supply to activate and deactivate the electrostatic or magneticadhesive force of the end effector.

The use of one, various interchanged, or a combination of end effectortechnologies is driven by the physical properties of the grasped objectsas to generate a sufficient amount of lift force to carry the objects bythe manipulator arm without causing damage or visible alterations.

The presently disclosed system design also includes an extension tool170 mounted at the end of the robot manipulator arm 120, as shown inFIG. 1. This tool 170 enables the manipulation robot 100 to position thegrasping end effector 175 at a sufficient depth into a storage rack sothat the end effector may reach a desired item while maintainingclearance for the manipulator arm 120 itself from contacting theinfrastructure. It also enables the end effector 175 to reach into shelfcorners where it would otherwise not have clearance for the robotmanipulator arm 120. In certain embodiments, the extension tool 170 maybe sized based on specifics of the logistics facility such as, forexample, shelf depth. That is, the extension tool 170 may be long enoughto reach into the back of a shelf, as mentioned above, so that the endeffector 175 may pick a piece placed therein. Furthermore, the extensiontool 170 may have a diameter that is smaller than the diameter of theend effector 175. This may allow the extension tool 170 to reach into ashelf without obstructing the view of the end effector 175 and/or thepiece to be picked, and may simplify the computation required to locatethe piece as no additional sensor information may be required to locatethe extension tool; information regarding the end effector 175 would besufficient.

In certain embodiments of the system, the extension tool 170 may be areplaceable or switchable part of the robot manipulator arm 120. In thisway, logistics facilities having different configurations such as, forexample, deeper shelving, may be accommodated by simply switching outthe extension tool 170 to one more suited for the work (e.g. a longerextension tool). Furthermore, in embodiments where the end effector 175is mounted directly to the extension tool 170, different end effectorsmay require different connection mechanisms at the end of the extensiontool. As such, the ability to switch out the extension tool 170 to onehaving a suitable connection means may improve the ease of use of themanipulation robot 100.

After pieces are picked, they may be placed into the storage bed 140 fortransportation. The bed may also carry a container 145, such as a box ortote, in which the items can be placed. This method enables multipleitems to be picked for a given order or batch of orders. This methodfrees the robot manipulator arm 120 to pick additional pieces withoutneeding to take multiple trips to and from an order transfer area 360(See FIG. 3). Additionally, by carrying a packing box or container ortransport tote 145 onboard, the manipulation robot 100 is able toaggregate order pieces together into a single container that can beeasily swapped with a different container for additional orderfulfillment.

In certain embodiments, the storage bed 140 may comprise a calibrationtarget which may be viewed by one or more sensors 110 placed at acentral location on the manipulator arm 120 (see FIG. 1). Thesecentrally located sensors 110 may be positioned to view the calibrationtarget on the storage bed 140 when the manipulator arm is rotated. Assuch, information on the calibration target may be used to calibratethese sensors 110 to ensure that all parameters are withinspecifications, and if not, update the parameters to reflect the currentconfiguration. The dual use of the storage bed 140—as a platform to holdpicked items and as a calibration target—reduces the size profile of thepresently disclosed manipulation robot and improves the accuracy of thesystem.

The central location of one or more sensors 110 on the manipulator arm120 allows for improved piece picking accuracy. First, these sensors 110will have an unobstructed and enlarged view of the items to be pickedand their storage locations, which is improved over the view that isprovided by sensors placed at the end of a robotic arm or on the mobilebase, as is the case in many prior art systems. Furthermore, sensorsmounted at the end of a robotic arm may get in the way of, or reduce therange of positions available to, an end effector. Sensors mounted on themobile base may have their view of the items to be picked and theirstorage locations impeded by movement of the robotic arm.

Second, the central location of the one or more sensors 110 on themanipulator arm 120 provides improved measurement accuracy of the itemto be picked. The high degree of freedom manipulator arm 120 may moveand articulate at more than one point along the arm, and each movementintroduces potential error to any measurements that may be made betweenthe end effector 175 and the items to be picked by sensors positioned onthe mobile base 160. Placement of the sensors 110 at a central locationmay reduce this error by bringing the sensors 110 closer to the items tobe picked, and thus removing the error inherent in several points ofarticulation.

FIG. 3 shows an exemplary top view floor plan of a section of alogistics facility 300. The presently disclosed system and method enableobject pieces to be picked within a defined robot work zone 330, whichstores stocked objects on common commodity shelving 310. The system andmethod may define a plurality of transfer areas 360 in which items wouldbe transferred to and from the manipulator robots 100. The transfer area360 may possibly interface with a packing and shipping station 350, or aconveyor 320 or a staging area, or any combination thereof.

At the transfer area 360, a worker may remove the picked items orcontainer 145 holding the items from the robot 100. If a container 145is removed, a new container 145 could be transferred to the robot 100for fulfillment of the next order. The method may employ transfer ofpicked items or the container 145 by a human operator or, in otherembodiments, the transfer of items may be automatic. That is, in someembodiments, the onboard robot storage bed 140 may have a mechanicallyactuated conveyance device that allows for automatic transfer. Theconveyance device may be a small conveyor belt or may be a set ofrollers or wheels, which is capable of shifting the held pieces orcontainer 145 to and from another platform or conveyance.

In an alternative embodiment, the automatic transfer of objects from theonboard storage bed 140 may be performed by the manipulator arm 120 ofthe manipulation robot 100. In such an embodiment, the manipulationrobot 100 may transfer individual pieces by using its end effector 175grasping mechanism or the robot may transfer a container 145 carried inthe storage bed by manipulating it with an extension tool 170 and endeffector tool 175. In any of these embodiments the system may bedesigned to interface automatically with a separate conveyor system 320which may be adjacent to the transfer area 360, whereby pieces orcontainers 145 could be automatically moved through a conveyor 360around a facility to and from a robot picking area 330. This method hasthe advantage of requiring less manual work to be done to transferobjects from a manipulator robot 100 after they are picked.

The system's central server 200 may be used to process order informationthat is transacted with a WMS 201, and may coordinate the fulfillment oforders with a plurality of manipulation robots 100. All computation onthe server 200 may be executed by one or more internal processors 220.In certain embodiments, the server may have two software modules thatenable this order fulfillment coordination. The first processor may be atask dispatch module 228, which analyzes orders received from a WMS 201,and determines which of the plurality of manipulation robots 100 is tobe assigned to an order. After a manipulation robot 100 is selected forpicking an order, the task dispatcher 228 instructs the robot 100 withhigh-level order picking information, such as, route navigation paths,SKU locations, and an order drop-off location. The task dispatcher 228works closely with a system state monitor 230 to obtain key feedbackinformation from the system. The system state monitor 230 maycommunicate with the manipulation robots 100 to keep track of theircurrent physical location within the facility, along with statusinformation, which may include but is not limited to: whether the robot100 is currently assigned an order, any faults or error modes, healthinformation, such as remaining battery power, or charging status.

The central server 200 may also be used to store and process centralizedSKU information in an SKU database 256, which stores informationrequired by the robots to complete the order picking. The processing ofthis SKU specific information is executed within a SKU analysis softwaremodule 226. The SKU information can include SKU size and shape data,which can include physical dimensions, 3D geometry that can includepoint and triangle mesh structures, item weight, appearance informationthat can include colorized texture maps, and may include SKU markingcodes, that can include barcode and UPC data. Additionally, the centralserver 200 may store information in the SKU database 256 about thelocations and regions on the surface of the individual SKU units thatare allowed, or not allowed, for grasping by the manipulation robot 100.This allows the manipulation robot 100 to grasp an item in a way that isknown to be safe and stable, and prevents the robot from grasping anitem at a point or in a way that is unsafe or unstable.

In certain embodiments, the onboard robot storage bed 140 may beconfigured to sense the weight of the piece placed thereon. Thisinformation may be communicated to the central server 200, and mayprovide additional verification that the correct SKU was picked, andthat the item was properly transferred to the onboard robot storage bed140. If the wrong weight is sensed in the storage bed 140, themanipulator arm 120 may be used to remove the item from the storage bed.The item may be replaced to the storage location by the manipulator arm120, or a signal may be sent to the central server 200 requesting manualassistance, such as from a human pick worker. In the event that noweight is sensed in the storage bed 140, the manipulator arm 120 may beused to select another replacement item and/or retrieve the droppeditem. Further, a signal may be sent to the central server 200 requestingmanual assistance, such as from a human pick worker, or to alert thesystem to a change in the SKU inventory.

The central server 200 can also store information about the state of theSKU inventory in the SKU database 256, and may process this informationin the SKU analysis module 226. Such information may include theposition of items in their stored location, the location and orientationof grasping points for the robot to attempt to pick, and the sequence inwhich items of the same SKU type and approximate location should bepicked from the shelf. This enables a sufficiently fast pickingoperation for the manipulation robot 100, such that picking geometry andsequencing can be planned and stored in memory 256 on the central server200 or, and also, on the local storage 216 and does not need to becomputed at the time of pick by a given manipulation robot 100. Thecentral server 200 enables multiple manipulation robots 100 to shareinformation about the state of inventory and plans for picking, so thatdifferent robots 100 can pick from the same storage location, withouteach one needing to sense and compute pick information.

Additionally, the central server 200 can store information about theinfrastructure of the facility of operation in a map storage database254. This can include information about the storage racks 310 such asshelving dimensions (width, depth and height), separate shelf levelheights, shelf face widths, and rack column widths. The infrastructureinformation can be created, modified and analyzed through a map creationsoftware module 224 on the central server 200. By using this module ahuman operator can manually create a facility map or may in someembodiments load the map data from a predefined file, such as a ComputerAided Drawing (CAD) file, or in other embodiments may load mapping dataautomatically collected by a robot 100, which can use its onboardsensors (150, 110), to observe the facility infrastructure andautomatically generate a map.

The manipulation robots 100 may have a set of sensors (150, 110) thatenable autonomous navigation within a facility and sensors 110 thatallow it to identify and localize individual SKUs for picking. Thesensors (150, 110) may be 3D depth cameras, color cameras, laser rangingdevices, or any combination thereof. These sensors (150, 110) in apreferred embodiment provide high resolution 3D point data to themanipulation robot 100 that details the presence of physical objectswithin their field of view. The sensors (150, 110) may be connected tothe onboard computer processor 218, which may process the 3D point andcolor data to extract information for navigation and picking. Indifferent embodiments, a unique set of sensors mounted on themanipulation robot 100 may be used for picking and for navigation. Themanipulation robot may be programmed to point the sensors in a directionthat is expected for the task.

In order to perform pick work, the manipulator robots 100 may move andnavigate between pick locations in the work zone 330 and an ordertransfer area 360. During navigation the sensor data may be processed bythe onboard computer processor 218 in a navigation software module 212to extract two modalities of information. The first modality may belocal mapping information that indicates which areas around themanipulation robot 100 are traversable and which areas containobstacles. The ground facing sensors 150 on the manipulation robot 100are primarily used to generate this mapping information. There may betwo ground facing sensors 150, a front-facing one and a rear-facing one.This unique design allows the manipulation robot 100 to navigate whiledriving both forwards and backwards, which in certain picking scenarios,eliminates the need for the manipulation robot 100 to turn around, thusreducing travel time and increasing picking efficiency.

The second sensor information modality may be visual or audible landmarklocations, such as the visual landmark marker 420 locations shown inFIG. 4, which presents a drawing view of a low-cost visual navigationmethod of the manipulation robot 100. In such an exemplary embodiment,the system may use landmarks such as visual markers 420, which may beplaced ahead of time in fixed locations around the facility ofoperation. The onboard sensors (150, 110) are used to detect thesemarkers 420 and locate the manipulation robot 100 relative to them. Thisenables the robot 100 to know precisely where it is in the facility.Each marker 420 may have a unique pattern that is different from othermarkers 420 within the same facility. The unique marker pattern 420 maybe recognized by navigation module 212 algorithms which may be run bythe onboard computer processor 218, thus allowing the manipulation robot100 to localize itself without ambiguity.

Exemplary landmarks include visual markers as described above, which mayinclude any identifiable unique visual pattern, such as bar codes,numbers, letters, geometric shapes, or even a specific pattern ofblinking lights, and audible markers, which may include at least uniquepatterns of sound or even specific tones of sound. Before a robot canuse landmarks for navigation, the characteristics of the landmarks maybe stored on the central server 200 or on the remote storage 216 of therobot. In a preferred embodiment, the characteristics of the landmarksare stored on the remote storage 216 of the manipulation robot 100 sothat the robot may navigate autonomously through a logistics facilityand may not require constant communication from the central server 200.

Additionally, in the embodiment depicted in FIG. 4, careful attention isgiven to the placement of markers 420, which may be vertically mountedon shelving 310. This allows the robot 100 to locate vertically mountedvisual markers 420, because they are within the field of view 410 of itsarm mounted sensor 110. Vertically mounted markers 420 are desirablebecause markers installed on the floor of a facility are more difficultto maintain. Floor markers are subject to tread damage from people andmachines within the facility and therefore need more frequentmaintenance.

In addition to onboard sensors (150, 110) and navigation software 212,the navigation process may also be aided by the central server 200. Theserver 200 may have access to the central facility map storage 254,which enables it to analyze the stored maps in depth and optimize routesbetween pick locations. As such, the central server 200 has a set ofroute planning algorithms in a software module 222 that allow it topre-compute navigation routes within the robot work zone 330, andbetween the work zone 330 and any transfer areas 360. These routes canbe stored after computation in a route storage database 252 within thecentral server 200, so that they can quickly be recalled andcommunicated to manipulation robots 100, for rapid response during orderfulfillment and for interchangeability between multiple robots 100.

To perform individual piece picks, onboard sensors (150, 110) may beused to detect and localize individual pieces along with the specificlocations and orientations of the grasp points 501 on the surface of apiece. FIG. 5 presents an exemplary diagram of how the manipulationrobot 100 may use its end effector 175 and extension tool 170, to graspa piece at a specific point and orientation 501. During a pick, thesensors 110 also locate the infrastructure around the pieces such asshelving levels 500, rack columns, shelf faces and signage. Thisgeometry information for the infrastructure is required for pickmanipulation algorithms in a manipulation software module 214 todetermine a pick trajectory that is collision free, such that themanipulation robot 100 is able to grasp the piece without colliding withsurrounding objects.

In a preferred embodiment, special attention has been given to theplacement of the picking perception sensors 110, which are mounted onthe manipulator arm 120 in an orientation that allows them to see thepick location while the end effector 175 is positioned above the storagebed 140, as is shown in FIG. 1B. This method enables the system tolocalize additional pick pieces, grasp positions and orientations aftera pick has been made and an item is being placed into the storage bed140. This picking geometry information can be stored in memory 256, onthe central server 200, or, and also, on the local storage 216 where itcan be recalled later to enable sufficiently fast picks of the same SKUthe next time it is required. Additionally, it is common for more than asingle item of a particular SKU to be picked for an order. In this case,the perception and localization computation of additional pick items canbe done at the same time the manipulator arm 120 is placing a previouspick in the storage bed 140, which may help to improve the speed andefficiency of picking multiple items which are in near proximity.

In some embodiments, an additional “fine tuning” sensor may be added tothe robot near the end effector tool 175 to help accurately perform apick grasp. After the picking sensor 110 positively identifies andlocalizes a pick point, there may still be some positioning errorpresent due to uncertainty in accuracy and calibration of the sensor110. Therefore, a tuning sensor may be mounted at the tip of the endeffector tool 175 on the robot 100 to more precisely locate the positionof the tool 175 relative to the desired pick location. The tuning sensorwould have the desired pick location in its field of view as the robotis attempting the pick grasp. As the manipulator arm 120 moves towardsthe desired pick location, the tuning sensor could be used to make smalladjustments that guide the tool 175 toward the desired point.

The manipulator robots 100 have a mobile base 160 that is controlled bythe onboard computer processor 218. The mobile base may have two maindrive wheels 167, each driven by a servo motor. Each drive wheel 167 mayhave an encoder that provides motion feedback, which is used toprecisely control the speed of each wheel in order to achieve thedesired rotation and translation velocities of the robot 100. Thefeedback data is also used for odometry to estimate the motion of therobot 100 relative to the facility. The odometry is responsible forguiding the robot 100 navigation at times when visual markers 420 areout of sensor (150, 110) range. The mobile base 160 may also use passivewheels, such as casters 165, for stability and weight distribution.

All systems onboard the manipulator robot 100 may be powered fromonboard batteries 190, which may be housed within the mobile base 160.The batteries 190 may supply power to the robot during navigation andpicking for a limited time, and may be rechargeable to maintainoperation through an economically viable work period. Battery chargingmay occur opportunistically during times at which no orders are presentfor the manipulation robot 100 to pick, or charging may occur separatelyfrom the manipulation robot 100. In this later case, the batteries 190may be swapped with separately charged batteries for continued operationof the robot 100.

For opportunistic charging, the manipulation robot 100 may have acharging station in a designated area of the facility 340 in which therobot 100 can make temporary electrical contacts which feed power intothe onboard batteries 190 while the robot 100 is present. For separatecharging, a battery hot-swap may be performed by using permanentlyinstalled smaller short-life (minutes) onboard batteries to maintainpower while a larger modular battery 190 is replaced with a fullycharged battery 190 of equivalent design. This prevents the manipulationrobot 100 from needing to power down during battery swap, which savestime. Hot-swapping may be done manually by a human operator, or may bedone automatically by internal mechanisms of the manipulation robot 100and charging station being used to physically swap batteries 190 whilethe robot 100 coordinates the procedure.

While specific embodiments of the invention have been described indetail, it should be appreciated by those skilled in the art thatvarious modifications and alternations and applications could bedeveloped in light of the overall teachings of the disclosure.Accordingly, the particular arrangements, systems, apparatuses, andmethods disclosed are meant to be illustrative only and not limiting asto the scope of the invention.

What is claimed is:
 1. A mobile manipulation robot comprising: a mobilebase; at least one manipulator arm having a first end portion pivotallycarried by the mobile base, a second end portion comprising an endeffector, and an extension tool positioned at or near the second endportion and configured to provide access to a piece without interferencefrom surrounding items or infrastructure within the logistics facility;a plurality of sensors; a remote communication interface; a memoryconfigured to store robot specific information; and one or more robotprocessors coupled to the plurality of sensors, the memory, the mobilebase, the remote communication interface, and the at least onemanipulator arm, wherein the memory comprises computer programinstructions executable by the one or more robot processors to processdata received from each of the plurality of sensors, and output controlsignals to the mobile base and the at least one manipulator arm, andwherein the plurality of sensors provide signals related to detection,identification, and location of the piece, and the one or more robotprocessors analyze the sensor signals to generate manipulator armcontrol signals to guide the extension tool and the end effector of theat least one manipulator arm to pick or put-away the piece with acollision free path throughout a controlled motion of the at least onemanipulator arm.
 2. The mobile manipulation robot of claim 1, whereinthe one or more robot processors are configured to receive at least oneorder comprising an identification for at least one piece to be pickedor put-away.
 3. The mobile manipulation robot of claim 2, wherein the atleast one order is received from a central server or a user inputdevice, wherein the user input device comprises a manual user interfaceon the mobile manipulation robot, an external swappable memory port onthe mobile manipulation robot, a proximity communication device, or acombination thereof.
 4. The mobile manipulation robot of claim 2,wherein the at least one order comprises a location within a logisticsfacility for the at least one piece to be picked or put-away, and aroute for the mobile manipulation robot to take within the logisticsfacility to pick or put-away the piece.
 5. The mobile manipulation robotof claim 2, wherein the identification for the piece comprises a shapeof the piece, a size of the piece, a weight of the piece, a color of thepiece, a property of construction material of the piece, a visualmarking on the piece, a barcode on the piece, or any combinationthereof.
 6. The mobile manipulation robot of claim 1, further comprisingat least one storage bed configured to sense a weight for the piece whenplaced therein.
 7. The mobile manipulation robot of claim 1, wherein theend effector is a suction cup which is connected to a vacuum pumpthrough a valve, wherein actuation of the valve is controlled by the oneor more robot processors.
 8. The mobile manipulation robot of claim 6,wherein the at least one storage bed comprises a platform, a pick-to-kitholder, a container holder, or a combination thereof.
 9. The mobilemanipulation robot of claim 1, wherein the mobile manipulation robot isconfigured to autonomously navigate and position itself within alogistics facility by recognition of at least one landmark by at leastone of the plurality of sensors.
 10. A mobile manipulation robotcomprising: a mobile base, at least one manipulator arm having an endeffector, a storage bed, a plurality of sensors, wherein at least onesensor is positioned on the at least one manipulator arm such thatrotation of the at least one manipulator arm directs the at least onesensor to view the storage bed, a remote communication interface, amemory configured to store robot specific information, and one or morerobot processors coupled to the plurality of sensors, the memory, themobile base, the remote communication interface, and the at least onemanipulator arm, wherein the memory comprises computer programinstructions executable by the one or more robot processors to processdata received from each of the plurality of sensors, and output controlsignals to the mobile base and the at least one manipulator arm, andwherein the plurality of sensors provide signals related to detection,identification, and location of a piece to be picked or put-away, andthe one or more robot processors analyze the sensor signals to generatemanipulator arm control signals to guide the end effector of the atleast one manipulator arm to pick or put-away the piece.
 11. The mobilemanipulation robot of claim 10, wherein the storage bed comprises acalibration target which allows calibration of the at least one sensorlocated on the at least one manipulator arm.
 12. A mobile manipulationrobot comprising: a mobile base, a vertical actuator stage configured toraise and lower relative to the mobile base, at least one manipulatorarm having a first end portion mounted on the vertical actuator stageand a second end portion comprising an end effector, a plurality ofsensors, a remote communication interface, a memory configured to storerobot specific information, and one or more robot processors coupled tothe plurality of sensors, the memory, the mobile base, the remotecommunication interface, and the at least one manipulator arm, whereinthe memory comprises computer program instructions executable by the oneor more robot processors to process data received from each of theplurality of sensors and output control signals to the mobile base andthe at least one manipulator arm, wherein the plurality of sensorsprovide signals related to detection, identification, and location of apiece to be picked or put-away, and the one or more robot processorsanalyze the sensor signals to generate manipulator arm control signalsto guide the end effector of the at least one manipulator arm to pick orput-away the piece.
 13. A method of piece picking or piece put-awaywithin a logistics facility, the method comprising: providing the mobilemanipulation robot of claim 1; generating an itinerary comprising atleast one order which includes an identification for at least one pieceto be picked or put-away, a location within the logistics facility forthe at least one piece to be picked or put-away, and a route for themobile manipulation robot to take within the logistics facility;receiving the itinerary at the one or more robot processors; moving themobile manipulation robot along the route; and picking the at least onepiece from the location or putting-away the at least one piece to thelocation.
 14. The method of claim 13, wherein moving the mobilemanipulation robot occurs autonomously by recognition of at least onelandmark by at least one of the plurality of sensors.
 15. The method ofclaim 13, wherein the at least one order is generated by a warehousemanagement system or a central server.
 16. The method of claim 13,further comprising, when picking the at least one item from thelocation: placing the at least one item on a storage bed of the mobilemanipulation robot; and sensing a weight for the at least one itemplaced on the storage bed.
 17. A method of piece picking or pieceput-away within a logistics facility, the method comprising: providingthe mobile manipulation robot of claim 10; generating an itinerarycomprising at least one order which includes an identification for atleast one piece to be picked or put-away, a location within thelogistics facility for the at least one piece to be picked or put-away,and a route for the mobile manipulation robot to take within thelogistics facility; receiving the itinerary at the one or more robotprocessors; moving the mobile manipulation robot along the route; andpicking the at least one piece from the location or putting-away the atleast one piece to the location.
 18. The method of claim 17, furthercomprising: rotating the at least one manipulator arm to direct the atleast one sensor positioned thereon to view a calibration target on thestorage bed; and calibrating the at least one sensor positioned on theat least one manipulator arm.
 19. A method of piece picking or pieceput-away within a logistics facility, the method comprising: providingthe mobile manipulation robot of claim 12; generating an itinerarycomprising at least one order which includes an identification for atleast one piece to be picked or put-away, a location within thelogistics facility for the at least one piece to be picked or put-away,and a route for the mobile manipulation robot to take within thelogistics facility; receiving the itinerary at the one or more robotprocessors; moving the mobile manipulation robot along the route; andpicking the at least one piece from the location or putting-away the atleast one piece to the location.
 20. The method of claim 19, furthercomprising: raising or lowering the vertical actuator stage of themobile manipulation robot to position the end effector of the at leastone manipulator arm closer to the at least one piece to be picked orput-away.